Job Analysis Validity 1 Running Head: PREDICTIVE VALIDITY OF A JOB ANALYSIS TOOL Exploring the Utility of Three Approaches to Validating a Job Analysis Tool

Building on prior research (Meyer, Foster, & Anderson, 2006), the present study examined the utility of three approaches to validate the Performance Improvement Characteristics (PIC; Hogan & Rybicki, 1998). The PIC is a job analysis instrument used to assess personal characteristics important to successful job performance. Hogan Personality Inventory (HPI; Hogan & Hogan, 1995) data in seven archival studies were weighted/modified in three different ways according to PIC profiles generated from the same jobs as well as other jobs and then correlated with overall performance. Meta-analytic evidence revealed that of the three different approaches (partialweighting, full-weighting, and profile similarity), the profile similarity approach was the best at differentiating jobs and predicting performance, thereby evidencing predictive validity of the PIC. Job Analysis Validity 3 Exploring the Utility of Three Approaches to Validating a Job Analysis Tool Neither the Uniform Guidelines on Employee Selection Procedures (Equal Employment Opportunity Commission, 1978; hereafter “Uniform Guidelines”) nor The Principles for the Validation and Use of Personnel Selection Procedures (Society for Industrial and Organizational Psychology, 2003; hereafter “Principles”) include stringent standards for the validation of personality-based job analysis tools. However, job analysis tools are often used to determine which selection instruments will predict job performance and serve, therefore, as the cornerstone of many validity studies. Therefore, it is important to demonstrate the validity of job analysis procedures in identifying the selection instruments that should be used for making personnel decisions. The present study explores three different approaches to validating job analysis tools, and identifies the merits and shortcomings of each. To control for variability between different job analysis methods, each validation approach was investigated using the same instrument: the Performance Improvement Characteristics inventory (PIC: Hogan & Rybicki, 1998). The PIC is a “worker-oriented job analysis method designed to evaluate personality-related job requirements” (J. Hogan & Rybicki, 1998). The PIC is used to identifyidentifies the personal characteristics necessary to perform a job, in contrast to many job analysis tools that are used to identify task and/or behaviorally related job characteristics (e.g., PAQ; McCormick, Jeanneret, & Mecham, 1972). The PIC’s structure was derived directly from the Hogan Personality Inventory (hereafter “HPI”; add in Hogan and Hogan, 1995 – the manual). The HPI is a rigorously studied and validated measure of normal personality used by many organizations as a selection and/or development tool. It has been modeled after the Five Factor Model of personality (Goldberg, 1990) and contains seven scales (see Table 1) which are mirrored by the PIC. Therefore, the PIC provides seamless Job Analysis Validity 4 translation of job analysis results into recommendations for which HPI scales to employ in selection systems. The PIC identifies the personal characteristics needed to execute successfully the requirements of a job and the degree to which possession of these personal characteristics improves job performance (J. Hogan & Rybicki, 1998). The PIC is completed by Subject Matter Experts (e.g., incumbents, supervisors; hereafter “SMEs”). The result is a profile that identifies the personality characteristics most critical to successful job performance. If the PIC is able to identify the required personal characteristics of a job, then we should see variability in the makeup of the PIC profiles across jobs. In practice, the PIC assists in understanding the requirements of a given job and which personality dimensions may be the best predictors of performance. The PIC is not used as a predictor itself. However, the PIC results are used to help identify which HPI scales should be used for the selection battery. In practice, the top three or four PIC scales from a job analysis are considered evidence toward the use of those HPI scales in the selection system. This evidence is combined with other sources of evidence such as meta-analytic results, transportability of validity, synthetic validity, and criterion-related validity to arrive at the final battery of HPI scales to be used. It is not possible to correlate PIC results with performance in an effort to validate the PIC, as it is only a job analysis tool that asks SMEs to respond in accordance with what the job requires rather than an assessment of their own personality. Instead, we must interject the PIC into the HPI-performance relationship to determine its validity. For the current study, we assess three different approaches to accomplish this: profile similarity, full-weighting, and partialweighting (later discussed in detail). When the HPI results for a given job are weighted based on Job Analysis Validity 5 the PIC profile from its own job analysis, the HPI should be predictive of performance. Further, the HPI should be more predictive of performance when weighted according to its own PIC profile than when weighted according to the PIC profile of another job; provided the two jobs differ in terms of the personal characteristics required to successfully perform. For example, bus driver performance should be better predicted by a battery of HPI scales selected based on the results of a bus driver job analysis than would a battery of HPI scales selected based on an accountant job analysis. The prior point requires further clarification. It is not our contention that each job will have a PIC profile that is completely unique and unlike any other. To the contrary, we assert that it is certainly possible to have two jobs that have disparate tasks and responsibilities yet require many of the same personal characteristics. Even further, we acknowledge that there should be a pattern across many jobs in which certain personal characteristics maintain their predictive power. Just as prior meta-analytic research has revealed that the Big Five Factors of Conscientiousness, Emotional Stability, and Agreeableness tend to predict performance across jobs (e.g., Barrick & Mount, 1991; Tett, Jackson, & Rothstein, 1991), we should also expect that the corresponding PIC scales of Prudence, Adjustment, and Interpersonal Sensitivity be commonly rated as important for successful performance. In fact, Hogan and Holland (2003) demonstrated meta-analytically that those three scales on the HPI do tend to predict performance (ρ = .36, .43, and .34, respectively). The first of the three validation approaches, profile similarity, employs a metric of similarity between predictor (HPI) scores and PIC scores, sometimes referred to as a profile correlation index (PCI: Timmerman, 1996). This approach is similar to that used by others to assess fit (e.g., Caldwell and O’Reilly, 1990). For this approach, the results of the PIC analysis are correlated with each incumbent’s HPI scores to determine the extent to which an individual’s Job Analysis Validity 6 HPI profile is congruent to the PIC profile for the job. The resultant correlation is then correlated with performance. As previously mentioned, the PIC is designed to assess the degree to which successful performance is associated with the various characteristics. Therefore, the extent to which an individual’s profile (HPI) is related to this model profile of successful performance (PIC) should be predictive of that individual’s success. Second, if the PIC does an adequate job of differentiating between jobs, there should be a stronger relationship between the PIC and performance when the PIC data from the target job is used than when PIC data from a different job is used. Hypothesis 1a: Across multiple jobs, profile correlation indices (PCI) will be positively and significantly related to performance. Hypothesis 1b: PCI’s will have a stronger positive relationship with performance when based on its relevant job analysis information than when based on a different job’s job analysis information. The second approach, full-weighting, utilizes an algorithm to weight all predictor scales according to their relative importance as identified by the PIC. Specifically, the percentage of total possible for each PIC scale is calculated. Then, each individual’s HPI scores are multiplied by these PIC percentages to arrive at weighted scale scores. The seven weighted predictor (HPI) scales were then summed and correlated with performance. This approach is similar to the one used by Arthur, Doverspike, and Barret (1996), in which job analysis information was used to weight various predictors for the purpose of creating a validated selection battery. As each HPI scale has been modified by PIC data intended to profile successful performance, the weighted sum should be correlated with performance. Further, it should be better predictive of Job Analysis Validity 7 performance when the weights are derived from its relative job analysis information (PIC) than from a different job. Hypothesis 2a: Across multiple jobs, full-weighted predictors will be positively and significantly related to performance. Hypothesis 2b: Full-weighted predictors will have a stronger positive relationship with performance when based on its relevant job analysis information than when based on a different job’s job analysis information The third approach, partial-weighting, was first used by Meyer, Foster, and Anderson (2006), in which a different algorithm is used that more closely mimics practice by weighting only the top three scales (based on PIC results) and eliminating the remaining scales, thereby negating their contribution to the overall predictor score. In practice, multiple HPI scales may be recommended for first-level screening cuts. Therefore, a weighting scheme was created that approximates common practice. Within each study (job), the PIC scales were rank-ordered based on the importance ratings provided by the PIC (percent of total possible). The highest ranked PIC scale was given a weight of “3”, the second highest a “2”, and the third highest a “1”. The remaining four PIC scales were weighted by “0”, thereby eliminating them from the subsequent analyses. The included scales are then summed and correlated with performance. Using this approach in the current study will provide a test of its generalizability to a new sample of jobs. Based on the positive results in Meyer, Foster, and Anderson (2006), we expect to find similar results in the present study. Hypothesis 3a: Across multiple jobs, partially-weighted predictors will be positively and significantly related to performance. Job Analysis Validity 8 Hypothesis 3b: Partially-weighted predictors will have a stronger positive relationship with performance when based on its relevant job analysis information than when based on a different job’s job analysis information. When considering which of the three approaches will be most effective in predicting performance and/or differentiating jobs, we have little reason to make a priori hypotheses. Considering that the partial-weighting approach only utilizes the top three scales, which more closely approximates practice, and it has been supported in past research (Meyer, Foster, & Anderson, 2006), we expect it to demonstrate similar utility. However, how this approach will compare to the other, previously untested approaches, is unclear. Job Analysis Validity 9